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ABSTRACT 



A cornerstone of the Kentucky Education Reform Act of 1990 
was the creation of a high- stakes performance assessment program called the 
Kentucky Instructional Results Information System (KIRIS) . KIRIS test results 
were the basis for granting monetary rewards to schools and school districts 
where student test performance improved significantly and for levying 
sanctions where student performance declined. KIRIS was based on the belief 
that school districts and educators should be held accountable regardless of 
the advantaged or disadvantaged circumstances of their communities. In 
contrast, recent research suggests that socioeconomic factors associated with 
geographic location may strongly influence on school system performance and, 
therefore, on test results. Using data from 176 Kentucky school districts, 
this paper analyzes the spatial distribution of socioeconomic factors and 
educational outcomes. Data include 1992-96 mean accountability scores, which 
reflect district performance at grades 4, 8, and 12; rural -urban location; 
median household income; percentage of students on free or reduced- cost 
lunch; teen birth rate; independent versus county school district; 
per-student spending; and enrollment. Accountability scores were strongly and 
negatively influenced by percentage of free/reduced lunch and were positively 
influenced by rurality. Socioeconomic factors had a spatial distribution that 
was spatially autocorrelated; these factors increased in importance at higher 
grade levels. Policy recommendations are offered. (Contains 15 references.) 
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******************************************************************************** 

♦ Reproductions supplied by EDRS are the best that can be made ♦ 

♦ from the original document. ♦ 

******************************************************************************** 



er|c 



0 2 2 0 4 7 



r- 

o^ 






s 



U.8. DEPARTMENT OF EDUCATION 

Office of Educational Research and improvement 

™ v‘‘^'ir:\he ’pef»n 

originating it 

□ Minor changes have been made to improve 

reproduction quality. 

# Points of view or opinions stated in 

ment do not necesMr»ly represent official 
OERI position or policy. 



•‘PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 




BiiH 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC). " 






A SPATIAL ANALYSIS OF CONTEXTUAL EFFECTS ON EDUCATIONAL 
ACCOUNTABILITY IN KENTUCKY 
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Morehead State University 
Morehead,KY 40351 



1. INTRODUCTION 

A call to make public education more effective by making it more accountable has swept the 
nation during the 1990s (Federal Reserve Bank of New York 1998, Ladd 1996). Kentucky 
became one of the states that led this movement when its legislature passed the ambitious 
Kentucky Educational Reform Act of 1990 (Guskey 1994). A cornerstone of this reform 
initiative was the creation of a high-stakes performance assessment program, called the Kentucky 
Instructional Results Information System (KIRIS). The KIRIS test results were used as a basis 
for granting monetary rewards to school and school districts that showed significant 
improvement in student test performance and for levying sanctions against schools and districts 
where performance declined. The philosophy gj -.iding the use of KIRIS was that all school 
systems in the state could effect steady and substantial improvement in KIRIS scores irrespective 
of the socioeconomic context in which the schools are found. In other words, the belief was 
maintained that school districts and educators should be held accountable regardless of the 
advantages or disadvantages with which their local conununities presented them. Against this 
philosophy recent research has begun to suggest that, at least in the long if not in the short run, 
socioeconomic factors associated with geographic location may have a strong determining 
influence on school system performance and therefore on the accountability test results (Reeves 
1998, Reeves 2000, Reeves and Grubb 1999). 

At this time, the accountability movement has given little attention to how much socioeconomic 
context influences educational outcomes. Furthermore, there is almost no recognition in the 
research literature that socioeconomic factors are spatially distributed and thus can be subjected 
to geographic analysis. The purpose of this investigation is to show how such an analysis might 
be done using Kentucky accountability results. 
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THE SAMPLE AND MEASUREMENTS 



The sample for this study consists of 176 Kentucky school districts. The measurements taken on 
each district consist of accountability index scores provided by the Kentucky Department of 
Education and various contextual influence measures derived from census reports and other 
sources. 

2. 1 . THE DEPENDENT VARIABLE 

For dependent variables, we used the 1 992-96 mean accountability scores for each Kentucky 
school district, obtained from the Kentucky Department of Education (KDE). The accountability 
score is a weighted sum of the following components: reading (14%), mathematics (14%), 
science (14%), social studies (14%), arts and humanities (7%), practical living (7%), writing 
(14%), and noncognitive data (16%). The noncognitive data are compiled from student 
attendance, retention, dropout, and transition to adult life. 

The scores reflect the district performance at the 4th, 8th, and 12th grades, as well as an overall 
composite score. The composite mean accountability scores ranged from 33.5 to 63.6 with an 
average of 41.9 and a standard deviation of 4.4. The 4th grade scores ranged from 31.1 to 63.6, 
averaged 41.9 and had a standard deviation of 5.2; 8th grade scores ranged from 30.6 to 61.1, 
averaged 41.5 and had a standard deviation of 4.9; and 12th grade scores ranged from 31.0 to 
58.8, averaged 41 .8 and had a standard deviation of 5.0. 

2.2. CONTEXTUAL MAIN EFFECTS 

To determine how rural-metro differences and other contextual effects would influence the high- 
stakes accountability scores, we devised the following independent variables: 

2.2.1. Rural-Metro Differences 

Khattri et al. (1997) have noted that it is still an open question if geographic location is as 
important a factor as poverty in contributing to educational outcomes. Of particular interest in 
this study is the combined effect of geographic location and population concentration on the 
accountability scores. We developed categorical variables expressing these rural-metro 
differences by recoding the 1993 Urban Influence Codes put out by the Economic Research 
Service of the U.S. Department of Agriculture. The Urban Influence Codes classify all U.S. 
counties into nine exclusive categories based on (1) the size of the Metropolitan Statistical Area 
(MSA) for metro counties and (2 categories) (2) adjacency to metro counties and size of the 
largest town for nonmetro counties (7 categories). We simplified this classification scheme, 
because Kentucky does not have any large metro counties and several of the metro-adjacent 
categories are either missing or little represented. 

Our recoding of the Urban Influence Codes resulted in the following categories: Metro counties 
{i.e., located in an MSA) became the reference category for the multivariate analysis. Twenty- 
four percent of the school districts in the sample are classified “metro”. Nonmetro counties were 
of two broad types, either they were adjacent to a metropolitan county in which case they are 
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referred to as “metro-adjacent” or they were not adjacent to a metropolitan county in which case 
they are referred to as “rural”. Metro-adjacent districts comprise 25% of the Kentucky sample. 
The rural counties are subdivided into three discrete categories according to the size of the largest 
town or city. “Rural 1” refers to a rural county in which the largest town has a population less 
than 2,500. Seventeen percent of the school districts are in this category. ^ “Rural 2” counties 
have a town with 2,500 to 9,999 inhabitants. Twenty-three percent of the school districts fall 
into this category. “Rural 3” is a category of rural counties in which the largest town has a 
population of 10,000 or more. This category contains eleven percent of the school districts. 

2.2.2. District Median Household Income 

Our measure of the SES level of the general population in the school district is the median 
household income. This information was acquired from the School District Data Book (National 
Center for Education Statistics) and is based on a special run of the 1990 U.S. Census data. For 
the present study, the median household income of the Kentucky school districts varied between 
$8,150 and $82,435 with a mean of $20,258 and a standard deviation of $7,581. 

2.2.3. Percentage of Students on Free/Reduced Lunch 

The mean percentage of students in the school district on the free or reduced-cost lunch program 
is the variable that we chose to measure student poverty. The measure was obtained by 
averaging KDE data for the 1989-90 and 1994-95 school years. This variable averages 44.9 for 
the entire state, with a standard deviation of 17.3. The mean percentages of students on 
free/reduced lunch vary from 2.3 to 89.1 percent. 

2.2.4. Teen Birth Rate 

Teen birth rate may be considered a proxy measure for the youth opportunity environment in the 
school district (Bickel et al. 1997) The teen birth rate is defined to be the number of births in a 
county per 1,000 females, aged 12 to 17 years. In this study, an average teen birth rate was 
calculated for each Kentucky county using 1992-94 data. We obtained these data from the 1995 
Kentucky KIDS COUNT. Across the state of Kentucky, the average teen birth rate is 20.9 with a 
standard deviation of 6.33. The county with the lowest teen birth rate had an average of 6.4 
while the county with the highest rate had an average of 39.0. 

2.2.5. Independent School District 

This variable, a dummy, denotes that the school district is one of 56 independent districts in 
Kentucky (32 percent of the total 176 districts). The independent districts tend to be smaller, 
wealthier, less rural, and higher performing on the accountability index than the county districts. 

2.2.6. Per Student Spending 

Although previous studies have found only a weak correlation between student performance and 
per student spending (e.g., Hanushek 1997), we thought it prudent to incorporate a measure of 
per student spending by school district as a control variable, since metropolitan school districts 



are often better ftinded than their rural counterparts--a pattern in Kentucky that is also found 
nationwide (Stem 1994). This variable was constracted by averaging KDE data for two school 
years, 1989-90 and 1994-95. The resulting measure finds the average level of per student 
spending statewide is $4,377 with a standard deviation of $518, while the range in spending is 
between $3,584 and $7,994. 

2.2.7. Enrollment 

Some researchers have argued that large schools and large districts are detrimental to 
disadvantaged students (Friedkin and Necochea 1988). Therefore, we chose as a third control 
variable the average student enrollment by school district. To obtain this variable we used the 
KDE 1994-95 enrollment figures for each Kentucky school district. Across the state school 
district enrollment varies greatly, from a tiny independent school district in a metropolitan 
coimty that has only 208 students to a metro coimty district with an enrollment of 34,165 
students. The mean enrollment statewide is 3,371 students with a standard deviation of 4,079. 

2.3. INTERACTION EFFECTS 

In addition to the above main effect variables, our regression model examined two types of 
interactions. In constructing these interaction terms, the continuous variables were centered on 
their means to reduce collinearity with the corresponding main effects. 

2.3.1. Enrollment x Percent of Students on Free/Reduced Limch 

The first of the interaction terms was the bilinear interaction between enrollment and percentage 
of students on free/reduced limch. We decided to test for this effect because Howley (1996) 
found that small district size has a beneficial moderating effect on the academic performance of 
low-SES students. 

2.3.2. Rural-Metro Categories X Percent of Students on Free/Reduced Lunch 

With the exception of studies by Alspaugh (1992) and Lippman et al. (1996) that focused on the 
rural-urban dichotomy, little research has been conducted on how geographic location moderates 
the effect of student poverty on educational outcomes. In the present study we have employed 
interaction terms to capture the combined effect of location and poverty. Four interaction terms 
were constructed to test for the effect of percent of students on free/reduced lunch when 
moderated by the different rural-metro categories. When assessing the interaction effects, the 
reference category is the interaction term, metro x % students on free/reduced lunch. 

3. REGRESSION ANALYSIS 

The regression analyses used in the present paper are based on Reeves and Grubb (1999). The 
regression models (see Table 1) present the standardized coefficients of the main effects and the 
interaction effects on the accountability score. In the composite model, the largest effect is the 
percent of students on free/reduced lunch. Median household income is not sigmficant, while 



er|c 



5 



teen birth rate marginally approaches significance in this model. All three categories of ruralness 
are significantly positive when compared with the metro category. The model also tests the 
effects on the accountability score of the percent of students on ffee/reduced lunch when 
moderated by geographic location. The results of the interaction effects are significantly positive 
for all rural categories. Furthermore, the more rural the district the greater the strength of the 
interaction effect. Overall, the composite model predicts nearly 49 percent of the variance. 

TABLE 1 

STANDARDIZED OLS REGRESSION COEFFICIENTS FOR VARIABLES PREDICTING 
1992-96 KENTUCKY SCHOOL DISTRICT ACCOUNTIBILITY SCORES BY GRADE 

LEVEL 



Variables 


COMPOSITE 


4 


GRADE 

8 


12 


Median household income 


.171 


.101 


.161 


-.033 


Percent ffee/reduced lunch 


-.906** 


-.668** 


-.915** 


-.819** 


Teen birth rate 


-.123- 


-.009 


-.163* 


-.192** 


Independent school district 


.208** 


-.051 


.186* 


.411** 


Per student spending 


.060 


.159- 


-mi 


.010 


Enrollment 


.045 


-.052 


.096 


.091 


Enrollment X % ffee/reduced lunch 


.057 


.044 


.051 


.009 


Rural 1 : town < 2,500 


.181- 


-.015 


.281** 


.148 


Rural 2: town 2,500-9,999 


.338** 


.169 


.453** 


.196* 


Rural 3: town > 10,000 


.256** 


.181- 


.284** 


.142- 


Metro-adjacent 


.216* 


.113 


.325** 


.078 


Rural 1 X % ffee/reduced lunch 


.349** 


.284* 


.345** 


.267* 


Rural 2 X % ffee/reduced lunch 


.202* 


.140 


.252** 


.116 


Rural 3 X % ffee/reduced lunch 


.146* 


.135 


.144^ 


.093 


Metro-adjacent X % ffee/reduced lunch 


.165- 


.121 


.211* 


.089 


R' 


.532 


.249 


.471 


.556 


Adjusted R^ 


.488 


.179 


.422 


.513 



~p < .10; *p < .05; **p < .01 . Note: For Composite, 4th and 8th grade models, N= 176; for 
12th grade model, V= 171. 



The grade-level models reveal a remarkable tendency. When the 4th and 8th grade models are 
compared, the predicted variance more than doubles with the higher grade level and increases 
again by 9 percent when the 8th and 12th grade models are compared. It should also be noted 
that teen birth rate has a significant negative effect on 8th and 12th grade scores but not on the 
scores of the 4th graders. Rural-metro differences are important at the 8th grade level but are 
comparatively less important at the 4th or 12th grades. Finally, independent school district has 
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an especially strong, positive effect on 12th grade scores. 



FIGURE 1 

COMPOSITE SCORES BY SCHOOL DISTRICT, 1992-1996 



Z-Scores 




a) Standardized KIRIS Scores 



Z-Scores 
r~1 -2.399 - -1 

rm -^-0 




b) Standardized Residuals 
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FIGURE 2 

GRADE 4 SCORES BY SCHOOL DISTRICT, 1992-1996 




a) Standardized KIRIS Scores 





b) Standardized Residuals 



FIGURE 3 

GRADE 8 SCORES BY SCHOOL DISTRICT, 1992-1996 




a) Standardized KIRIS Scores 



Z^cores 





b) Standardized Residuals 




FIGURE 4 

GRADE 12 SCORES BY SCHOOL DISTRICT, 1992-1996 



Z-Scores 
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a) Standardized KIRIS Scores 



ZrScores 




b) Standardized Residuals 
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4. 



METHODS OF RESIDUAL ANALYSIS 



In order to measure the amount of clustering in KIRIS scores, a measure of spatial 
autocorrelation is needed. One measure useful in determining the level of spatial autocorrelation 
in variables measured at the interval/ratio level is Moran’s /-coefficient. Moran’s / uses the form 
of the typical correlation coefficient but compares neighboring areal units (Griffith and Amrhein 
1991). Since the current data are standardized scores, the mean is zero, simplifying the standard 
Moran formula to: 

I = [n/(SiSj Cij)] [SiSj (Zi Cij Zj)/Si z^'] [1] 

where, n = number of areal units; 

cij = 1 if areal unit / is adjacent to areal unit j, 0 otherwise; 
and zj (zj) = standardized score for areal unit / (j). 

The results of Moran’s / tend to range between -1 and 1 like the traditional correlation 
coefficient, though they are not limited to that range. More positive values indicate positive 
spatial autocorrelation in which similar values are clustered, more negative values indicate 
negative spatial autocorrelation in which dissimilar values are near each other spatially. 

The expected value of Moran’s / in a sample and standard error are (Griffith and Amrhein 1991): 



E(I) = -[1 + n SiSj Zi Cij z/(n-l)SiSj CiJ/(n-2) 


[2] 


s, = (2/SiSjCij)''^ 


[3] 



Figures 1-4 show the patterns of a) standardized KIRIS scores for Composite, Grade 4, Grade 8 
and Grade 12, respectively, and b) standardized residuals for Composite, Grade 4, Grade 8 and 
Grade 12, respectively. Table 2 indicates the Moran’s /, E(/), cr„ and z-value of the standardized 
KIRIS scores and of the standardized residuals of the model. 



5. DISCUSSION 

Table 2 indicates the value for Moran’s / as calculated from the results presented graphically in 
Figures 1-4. The range for Moran’s / is similar to that of the traditional correlation coefficient. 
Columns 2-5 represent the level of spatial autocorrelation of standardized KIRIS scores. 

Columns 6-9 represent the level of spatial autocorrelation of the standardized residuals when 
contextual effects are held constant at their mean value or at the reference category. 

Figures 1-4 demonstrate the results in Table 2. For instance, in Figure la there is a cluster of 
poorly performing districts in the southeastern part of the state and a cluster of highly performing 
districts in the north central region. This represents a relatively high degree of spatial 
autocorrelation (/= 0.182). In Figure lb, the large clusters are removed in large part, though 
some clusters of high residuals are created for instance in the northeast part of the state. This 
represents a moderate degree of spatial autocorrelation (/= 0.105). Figure 4 provides an even 
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starker contrast. Figure 4a demonstrates a large cluster of poor performance in the eastern and 
southeastern portion of the state and clusters of high performance in the central and western 
portions (/= 0.208). Figure 4b illustrates no discernible clusters with the lowest level of spatial 
autocorrelation of all the distributions measured in this paper (/= 0.062). 



TABLE 2 

MEASURES OF SPATIAL AUTOCORRELATION 





KIRIS Scores 


Standardized Residuals 




Comp 


Grade 4 


Grade 8 


Grade 12 


Comp 


Grade 4 


Grade 8 Grade 12 


I 


0.182 


0.087 


0.178 


0.208 


0.105 


0.070 


0.080 


0.062 


E(I) 


-0.007 


-0.006 


-0.007 


-0.007 


-0.006 


-0.006 


-0.006 


-0.006 


a(I) 


0.051 


0.051 


0.051 


0.051 


0.051 


0.051 


0.051 


0.051 


z 


3.727 


1.845 


3.640 


4.234 


2.194 


1.510 


1.690 


1.345 



Several general results are obvious from the table and figures. First, in every case, the use of a 
regression model to account for variation in contextual effects helps to diminish the level of 
spatial autocorrelation in the residuals as compared to the uncorrected scores. Clearly, contextual 
effects are spatially autocorrelated, so most of the autocorrelation of the KIRIS scores is 
associated with autocorrelation of the contextual variables. 

Second, the value of Moran’s / of the residuals is consistent at a low score for all three of the 
grade-specific cases. In all three cases, it is not significant at the level of qf= 0.05. This suggests 
that the regression model helps to eliminate spatial autocorrelation in the KIRIS scores. 

Third, for the KIRIS scores, as grade level increases, the level of spatial autocorrelation 
increases. This suggests that contextual effects are more important as grade level increases since 
the spatial autocorrelation can be removed by accounting for contextual effects in a regression 
model. Therefore, it seems that students become more influenced by the contextual effects of 
their communities as they get older. 



6. CONCLUSIONS 

From this analysis, it seems clear that contextual effects have an influence on educational 
outcome. In particular, this study has shown that socioeconomic factors have a spatial 
distribution that is spatially autocorrelated. Furthermore, it seems that the importance of these 
factors in student achievement increases with higher grade levels and thus is more influential for 
middle and high schools than for elementary schools. 

Two policy prescriptions emerge from this analysis. First, socioeconomic factors should be 
factored into the assessment program for high schools and middle schools. The current 
uncorrected KIRIS scores overstate the performance of schools and districts in advantageous 
situations and understate the performance of schools and districts that are disadvantaged. 



Second, the geographical context also needs to be taken into account. The current method of 
assessment ignores the spatial distribution of KIRIS scores. Neighboring districts have similar 
values that cannot be explained by district or school performance. If spatial autocorrelation is 
present in a dataset, a spatially varying explanatory variable has been left out of the analysis. 
When nearby districts seem to be performing similarly without any causal explanation as to how 
or why, the assessment is misleading. 

This study underscores the importance of considering socioeconomic context and geographic 
effects when assessing schools. Further research into the role that spatial analysis can play in 
gaining a better understanding of educational performance is needed. In particular, the pattern of 
accoimtability scores at the school level would provide a needed refinement to the current 
research. Unfortunately, the areas from which individual schools draw students are constantly 
changing, making the collection of contextual data difficult. 
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